Papers with open-world tasks
VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown significant promise in embodied decision-making tasks within virtual open-world environments, but lack domain-specific knowledge. |
| Approach: | They propose a cost-effective agent framework that integrates cross-modal domain knowledge and finetunes a dedicated object detection model for visual analysis. |
| Outcome: | The proposed framework reduces the requirement for domain-specific training data from millions of samples to a few hundred. |
Meta-Tool: Unleash Open-World Function Calling Capabilities of General-Purpose Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models struggle with addressing diverse user inquiries in open-world tasks. |
| Approach: | They propose a plug-and-play tool retrieval system for LLMs to access external tool library and use retrieved tools to solve user's problem. |
| Outcome: | The proposed model improves on a finetuned version of LLaMA-3.1 and 2,800 dialogues and 7,361 tools spanning ten distinct test categories. |